Depth camera light leakage avoidance

ABSTRACT

Disclosed are a device and a method of depth sensing that handle light leakage issues. In some embodiments, the depth sensing device includes a light emitter that illuminates an environment of the depth sensing device. The device identifies a first portion of the emitted light that is prevented from reaching the environment of the device due to being redirected by an optical component located in proximity to the light emitter. An imaging sensor of the device detects a second portion of the emitted light that reaches and is reflected by a surface in the environment of the device other than a surface of the optical component. The device generates, based on the second portion of the emitted light, a depth map that includes a plurality of values corresponding to distances relative to the device, wherein said generating excludes from consideration the identified first portion of the emitted light.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application for patent is a Continuation of U.S. patent applicationSer. No. 15/428,078, titled “DEPTH CAMERA LIGHT LEAKAGE AVOIDANCE”,filed on Feb. 8, 2017, the entire contents of which is incorporated byreference herein.

BACKGROUND

Depth sensing technology can be used to determine a person's location inrelation to nearby objects or to generate an image of a person'simmediate environment in three dimensions (3D). One application in whichdepth (distance) sensing technology may be used is in head-mounteddisplay (HMD) devices and other types of near-eye display (NED) devices.Depth sensing technology can employ a time-of-flight (ToF) depth camera.With ToF based depth sensing technology, a light source emits light intoits nearby environment, and a ToF camera captures the light after itreflects off nearby objects. The time taken for the light to travel fromthe light source and to reflect back from an object to the ToF camera isconverted, based on the known speed of light, into a depth measurement(i.e., the distance to the object). Such a measurement can be processedwith other similar measurements to create a map of physical surfaces inthe user's environment (called a depth image or depth map) and, ifdesired, to render a 3D image of the user's environment.

SUMMARY

Introduced here are a device and a method (collectively andindividually, “the technique introduced here”) of depth sensing. In someembodiments, the depth sensing device includes a light emitter (alsoreferred to as illumination module) and an imaging sensor. The lightemitter illuminates an environment of the depth sensing device. Thedepth sensing device identifies a first portion of the emitted lightthat is prevented from reaching the environment of the depth sensingdevice due to being redirected by an optical component located inproximity to the light emitter. An imaging sensor of the depth sensingdevice detects a second portion of the emitted light that reaches and isreflected by a surface in the environment of the depth sensing deviceother than a surface of the optical component. The depth sensing devicegenerates, based on the second portion of the emitted light, a depth mapthat includes a plurality of values corresponding to distances relativeto the depth sensing device, wherein said generating excludes fromconsideration the identified first portion of the emitted light.

In some embodiments, the depth sensing device includes an illuminationmodule, an optical component located in proximity to the illuminationmodule, an imaging sensor, and a process. The illumination module emitslight. A first portion of the emitted light illuminates an environmentof the depth sensing device. The optical component redirects andprevents a second portion of the emitted light from reaching theenvironment of the depth sensing device. The imaging sensor receives thefirst and second portions of the emitted light and records an imagebased on the received light. The processor generates a processed imageby subtracting a light leakage mask from the recorded image. The lightleakage mask includes pixel values corresponding to the second portionof the emitted light that is prevented from reaching the environment ofthe depth sensing device due to being redirected by the opticalcomponent. The processor converts the processed image into a depth mapthat includes a plurality of pixel values corresponding to depths of theenvironment relative to the depth sensing device

In some embodiments, the depth sensing device includes an opticalcomponent, an illumination module, and an imaging sensor. Theillumination module is located in proximity to the optical component.The illumination module emits light towards an environment of the depthsensing device. A first portion of the emitted light is prevented fromreaching the environment due to being redirected by the opticalcomponent. A second portion of the emitted light illuminates theenvironment and is reflected through the optical component by a surfacein the environment other than a surface of the optical component. Theimagining sensor includes a shutter. The shutter closes during a firsttime period when the first portion of the emitted light redirected bythe optical component is reaching the shutter. The shutter opens duringa second time period when the imaging sensor receives through theoptical component the second portion of the emitted light reflected bythe surface of the environment. In some other embodiments, the shutteris not completely closed but is substantially closed, during which asubstantially reduced portion of the first portion of the emitted lightreaches the shutter.

Other aspects of the disclosed embodiments will be apparent from theaccompanying figures and detailed description.

This Summary is provided to introduce a selection of concepts in asimplified form that are further explained below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present disclosure are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 shows an example of an environment in which a virtual reality(VR) or augmented reality (AR) enabled HMD device can be used.

FIG. 2 illustrates a perspective view of an example of an HMD device.

FIG. 3 shows a front view of a portion of a sensor assembly of an HMDdevice.

FIG. 4A schematically illustrates an active depth sensing device withouta protective optical component.

FIG. 4B schematically illustrates an active depth sensing device with aprotective optical component.

FIG. 5 is a flow diagram illustrating a sample process of generating alight leakage mask during camera calibration.

FIG. 6 is a flow diagram illustrating a sample process of generating alight leakage mask on-the-fly.

FIG. 7 is a flow diagram illustrating another sample process ofgenerating a light leakage mask on-the-fly.

FIG. 8 shows intensities observed in four shutter images.

FIG. 9A shows shutter windows of a pulse-based ToF depth camera that arenot adjusted to avoid capturing leaked light.

FIG. 9B shows shutter windows of a pulse-based ToF depth camera that areadjusted to avoid capturing leaked light.

FIG. 10 shows a high-level example of a hardware architecture of asystem that can be used to implement any one or more of the functionalcomponents described herein.

DETAILED DESCRIPTION

In this description, references to “an embodiment,” “one embodiment” orthe like mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodimentintroduced here. Occurrences of such phrases in this specification donot necessarily all refer to the same embodiment. On the other hand, theembodiments referred to also are not necessarily mutually exclusive.

Some depth sensing devices such as HMD devices and other types of NEDdevices include depth cameras to detect depth information relating toobjects in the environment in which the device is located. The depthsensing device can include an illumination module (e.g., an LED or alaser) that actively casts light into the environment of the device.With the illumination module, the depth sensing device is also called anactive depth sensing device, and the depth camera is also called activedepth camera. One type of active depth cameras is a ToF camera.

The depth sensing device can further include a protective opticalcomponent (e.g., a visor or a transparent shield) that is placed infront of the depth camera, such as used in some HMD devices. Theprotective optical component protects the depth camera from physicaldamage. A depth sensing device with such an optical component can alsobe visually more appealing to consumers. However, such a protectiveoptical component tends to create a light leakage problem for the depthcamera, which is due to unintended light redirection by the opticalcomponent.

For example, the illumination module is designed to emit light thatilluminates the environment. The environment reflects the light and thedepth camera receives at least some of the reflected light for depthsensing. However, due to manufacturing imperfection of the protectiveoptical component or foreign objects (e.g., dust, a smudge orfingerprint) on the optical component, the optical component redirects aportion of the emitted light back to the imaging sensor of the depthcamera. In other words, the optical component prevents that portion ofthe emitted light (also referred to as leaked light) from reaching theenvironment. This phenomenon of light redirection due to the protectiveoptical component is called light leakage.

The leaked light is redirected by the optical component and does notreach the environment. As a result, the leaked light corresponds todepth values of the optical component instead of the environment. Sincethe imaging sensor of the depth camera receives both the leaked lightand the light reflected by a surface of the environment, the resultingcalculated depth values correspond to points that are between theoptical component and the environment. In other words, some of themeasured depth values no longer accurately represent the distancesbetween the environment and the depth sensing device.

To address the light leakage issue, the depth sensing device identifiesa first portion of the emitted light that is prevented from reaching theenvironment of the depth sensing device due to being redirected by theoptical component, and a second portion of the emitted light thatreaches and is reflected by the surface of the environment of the depthsensing device. The imaging sensor of the depth sensing device generatesbased on the second portion of the emitted light a depth map, whichincludes values corresponding to distances relative to the depth sensingdevice. The generation of the depth map excludes from consideration theidentified first portion of the emitted light, and therefore preventsinaccurate depth measurement due to the light leakage.

To achieve accurate depth readings, the first and second portions of theemitted light can be identified in various ways. In some embodiments,the depth sensing device includes an electronic shutter that opens onlywhen the second portion of the emitted light reaches the imaging sensor.In some other embodiments, the depth sensing device generates a lightleakage mask and subtracts the light leakage mask from the depth map.

FIGS. 1 through 10 and related text describe certain embodiments of atechnology for depth sensing. However, the disclosed embodiments are notlimited to NED systems or HMD devices and have a variety of possibleapplications, such as in computer monitor systems, head-up display (HUD)systems, self-driving automobile systems, information input systems, andvideo game systems. All such applications, improvements, ormodifications are considered within the scope of the concepts disclosedhere.

HMD Device Hardware

FIG. 1 schematically shows an example of an environment in which an HMDdevice can be used. In the illustrated example, the HMD device 10 isconfigured to communicate data to and from an external processing system12 through a connection 14, which can be a wired connection, a wirelessconnection, or a combination thereof. In other use cases, however, theHMD device 10 may operate as a standalone device. The connection 14 canbe configured to carry any kind of data, such as image data (e.g., stillimages and/or full-motion video, including 2D and 3D images), audio,multimedia, voice, and/or any other type(s) of data. The processingsystem 12 may be, for example, a game console, personal computer, tabletcomputer, smartphone, or other type of processing device. The connection14 can be, for example, a universal serial bus (USB) connection, Wi-Ficonnection, Bluetooth or Bluetooth Low Energy (BLE) connection, Ethernetconnection, cable connection, digital subscriber line (DSL) connection,cellular connection (e.g., 3G, LTE/4G or 5G), or the like, or acombination thereof. Additionally, the processing system 12 maycommunicate with one or more other processing systems 16 via a network18, which may be or include, for example, a local area network (LAN), awide area network (WAN), an intranet, a metropolitan area network (MAN),the global Internet, or combinations thereof.

FIG. 2 shows a perspective view of an HMD device 20 that can incorporatethe features being introduced here, according to certain embodiments.The HMD device 20 can be an embodiment of the HMD device 10 of FIG. 1.The HMD device 20 has a protective sealed visor assembly 22 (hereafterthe “visor assembly 22”) that includes a chassis 24. The chassis 24 isthe structural component by which display elements, optics, sensors andelectronics are coupled to the rest of the HMD device 20. The chassis 24can be formed of molded plastic, lightweight metal alloy, or polymer,for example.

The visor assembly 22 includes left and right AR displays 26-1 and 26-2,respectively. The AR displays 26-1 and 26-2 are configured to displayimages overlaid on the user's view of the real-world environment, forexample, by projecting light into the user's eyes. Left and right sidearms 28-1 and 28-2, respectively, are structures that attach to thechassis 24 at the left and right open ends of the chassis 24,respectively, via flexible or rigid fastening mechanisms (including oneor more clamps, hinges, etc.). The HMD device 20 includes an adjustableheadband (or other type of head fitting) 30, attached to the side arms28-1 and 28-2, by which the HMD device 20 can be worn on the user'shead.

The chassis 24 may include various fixtures (e.g., screw holes, raisedflat surfaces, etc.) to which a sensor assembly 32 and other componentscan be attached. In some embodiments the sensor assembly 32 is containedwithin the visor assembly 22 and mounted to an interior surface of thechassis 24 via a lightweight metal frame (not shown). A circuit board(not shown in FIG. 2) bearing electronics components of the HMD 20(e.g., microprocessor, memory) can also be mounted to the chassis 24within the visor assembly 22.

The sensor assembly 32 includes a depth camera 34 and an illuminationmodule 36 of a depth imaging system. The illumination module 36 emitslight to illuminate a scene. Some of the light reflects off surfaces ofobjects in the scene, and returns back to the imaging camera 34. In someembodiments such as an active stereo system, the assembly can includetwo or more cameras. In some embodiments, the illumination modules 36and the depth cameras 34 can be separate units that are connected by aflexible printed circuit or other data communication interfaces. Thedepth camera 34 captures the reflected light that includes at least aportion of the light from the illumination module 36.

The “light” emitted from the illumination module 36 is electromagneticradiation suitable for depth sensing and should not directly interferewith the user's view of the real world. As such, the light emitted fromthe illumination module 36 is typically not part of the human-visiblespectrum. Examples of the emitted light include infrared (IR) light tomake the illumination unobtrusive. Sources of the light emitted by theillumination module 36 may include LEDs such as super-luminescent LEDs,laser diodes, or any other semiconductor-based light source withsufficient power output.

The depth camera 34 may be or include any imaging sensor configured tocapture light emitted by an illumination module 36. The depth camera 34may include a lens that gathers reflected light and images theenvironment onto the imaging sensor. An optical bandpass filter may beused to pass only the light with the same wavelength as the lightemitted by the illumination module 36. For example, in a structuredlight depth imaging system, each pixel of the depth camera 34 may usetriangulation to determine the distance to objects in the scene. Any ofvarious approaches known to persons skilled in the art can be used formaking the corresponding depth calculations.

The HMD device 20 includes electronics circuitry (not shown in FIG. 2)to control the operations of the depth camera 34 and the illuminationmodule 36, and to perform associated data processing functions. Thecircuitry may include, for example, one or more processors and one ormore memories. As a result, the HMD device 20 can provide surfacereconstruction to model the user's environment, or can be used as asensor to receive human interaction information. With such aconfiguration, images generated by the HMD device 20 can be properlyoverlaid on the user's 3D view of the real world to provide a so-calledaugmented reality. Note that in other embodiments the aforementionedcomponents may be located in different locations on the HMD device 20.Additionally, some embodiments may omit some of the aforementionedcomponents and/or may include additional components not discussed abovenor shown in FIG. 2. In some alternative embodiments, the aforementioneddepth imaging system can be included in devices that are not HMDdevices. For example, depth imaging systems can be used in motionsensing input devices for computers or game consoles, automotive sensingdevices, earth topography detectors, robots, etc.

FIG. 3 shows a portion of the sensor assembly 32 of the HMD device 20,according to at least one embodiment. In particular, the sensor assembly32 includes sensors and electronics mounted to a circuit board 38, whichcan be mounted to the chassis 24 as mentioned above. The sensors mountedto the circuit board 38 include the depth camera 34 and the illuminationmodules 36-1 through 36-4. Other sensors that may be included in thesensor assembly 32 but are not shown in the figures or discussed furthermay include head-tracking cameras, visible spectrum cameras, ambientlight sensors, and the like. Some or all of these other sensors may alsobe mounted to the sensor assembly 32.

In the illustrated embodiment, illumination modules 36-1 and 36-2 arepositioned such that they emit light in slightly outwardly divergentdirections with respect to the depth camera 34; whereas illuminationmodules 36-3 and 36-4 are positioned such that they emit light directlyforward (i.e., parallel to the user's head-pointing vector). Moreover,illumination from illumination modules 36-3 and 36-4 has a reduced fieldof illumination and increased range from the depth camera 34 that isgreater than the range of illumination from illumination modules 36-1and 36-2 from the depth camera 34. Hence, illumination modules 36 arecollectively configured to illuminate the user's field of view, althoughthe illumination may not be visible to the user. The locations andpositions of the illumination modules 36 and the depth camera 34relative to each other as shown in FIG. 3 are merely examples of aconfiguration used for depth sensing; other configurations are possiblein the context of the technique introduced here.

Light Leakage Due to Protective Optical Component

FIG. 4A illustrates an active depth sensing device without a protectiveoptical component. The active depth sensing device 400A (e.g., HMDdevice 20) includes an illumination module 436 (e.g., an LED or a laser)and a depth camera 434. There is no protective optical component (e.g.,a visor) in front of the depth camera 434. The illumination module 436emits a light beam 450 towards a point 490 of the environment. The point490 reflects the light (as light beam 452) towards the depth camera 434.

The depth camera 434 captures the light beam 452 and determines the timetaken for the light to travel from the illumination module 436 to thepoint 490 and the time taken to travel from the point 490 to the depthcamera 434. The depth sensing device 400A converts the total time into adepth measurement as the relative distance to the point 490.

FIG. 4B illustrates an active depth sensing device with a protectiveoptical component. The active depth sensing device 400B (e.g., HMDdevice 20) includes an illumination module 436 (e.g., an LED or alaser), a depth camera 434 and a protective optical component 422 (e.g.,visor 22) in front of the depth camera 434. The illumination module 436is located in proximity to the optical component 422. In someembodiments, the distance between the illumination module 436 and theoptical component 422 is in the range of few millimeters (mm) to a few(e.g., 1-3) centimeters (cm), although in other embodiments thatdistance can be larger, and perhaps significantly larger Theillumination module 436 emits a light beam 450 towards a point 490 ofthe environment through the optical component 422. The point 490reflects the light (as light beam 452) towards the depth camera 434through the optical component 422.

However, there can be natural scattering from the optical component ormanufacturing imperfections in the optical component 422 or foreignobjects (e.g., dust, a smudge or fingerprint) on the optical component422. Due to the imperfections or foreign objects, the optical component422 can operate as a waveguide and change the direction (i.e., redirect)of a portion of the emitted light. For example, the illumination module426 emits a light beam 460 towards the optical component 422. Theoptical component 422, as a waveguide, receives the light beam 460 andguides the light to travel in the optical component 422 as light beam462. Then the optical component 422 redirects the light towards thedepth camera 434 as light beam 464.

The portion of emitted light that travels as light beams 460, 462 and464 is called leaked light. Since the optical component 422 prevents theleaked light from reaching the environment, the leaked light onlyreaches the optical component 422 and is redirected to the depth camera434.

The depth camera 434 receives the light beam 452 reflected by theenvironment point 490 (which is a point on a surface of theenvironment), as well as the light beam 464 redirected by the opticalcomponent 422. The total light travel time of the light beams 450 and452 corresponds to a distance between the environment point 490 and thedepth camera 434. In contrast, the total light travel time of the lightbeams 460, 462 and 464 corresponds to a distance between the opticalcomponent 422 and the depth camera 434.

As a result, by measuring the depth based on a combination of the lightbeams 452 and 464, the depth value measured by the depth camera 434deviates from the actual distance of the environment point 490. Thedeviation is called depth error. The resulting depth value is betweenthe distance of the environment point 490 and the distance of theoptical component 422. In other words, the leaked light due to theoptical component 422 causes an inaccurate depth measurement. The 3Dreconstruction of the environment based on the depth map is thereforedistorted.

The depth error also depends on the reflectivity of the environment. Forexample, if the environment point 490 has a relatively highreflectivity, the depth camera 434 receives more photons from the lightbeam 452, which corresponds to the correct depth value. Thus, the effectof depth error due to the light beam 464 is relatively low. On the otherhand, if the environment point 490 has a relatively low reflectivity,the depth camera 434 receives less photons from the light beam 452,which corresponds to the correct depth value. Thus, the effect of deptherror due to the light beam 464 is relatively high. Since theenvironment can have portions having different reflectivity levels, thedepth error due to the optical component 422 cannot be estimated simplyas corresponding to a constant percentage of the total received lightintensity.

Reduction of Depth Error Based on a Light Leakage Mask

To reduce or eliminate the depth error caused by the light leakage fromthe optical component, in some embodiments, the depth sensing devicegenerates a light leakage mask and adjusts the depth map by subtractingthe light leakage mask from the depth map. The light leakage maskincludes pixel values corresponding to light that is prevented fromreaching the environment of the depth sensing device due to beingredirected by the optical component.

As shown in FIG. 4B, the depth camera receives light signals from twodifferent light paths. A first portion of the received light isredirected by the optical component (e.g., visor) and corresponds to thedepth error. A second portion of the received light is reflected by asurface of an object whose depth is to be measured, e.g., a point of theenvironment of the depth camera. The light signals of the second portionchange when the depth sensing device or the object moves. In contrast,the light signals of the first portion (also referred to as lightleakage signals) remain constant when the depth sensing device or theobject moves, because the distance between the depth camera and theoptical component remains fixed.

The imaging sensor of the depth sensing device can record the lightleakage signals collectively as an image called light leakage mask.Because the light leakage mask remains constant, the depth sensingdevice can subtract the light leakage mask from a recorded image toderive a processed image. The recorded image includes light signals fromboth the first and second portions of the received light signals, whilethe processed image only includes light signals from the second portionof the light reflected by the surface of the object to be measured.

The depth sensing device then converts the processed image into a depthmap, which includes depth pixel values corresponding to distances of theobject or the environment, without depth error due to the light leakage.

The light leakage mask can be applied to various types of depth sensingsystems. For example, the light leakage mask can be applied to apulse-based time-of-flight depth camera, a phase-based time-of-flightdepth camera, or a structured light stereo matching system.

In the case of pulse-based ToF depth camera, the inaccurate depth valuesdue to the light leakage are always smaller than the correct depthvalues. In other words, the depth errors are always negative forpulse-based ToF depth camera. The reason is that the total travel timefor the light redirected by the optical component is smaller than thetotal travel time for the light reflected by a surface of theenvironment.

In a phase-based ToF depth camera, the illumination module (e.g., laser)and the shutter are frequency-modulated. The depth information isreconstructed by computing the phase difference between thecorresponding emitted and received light signals. For the phase-basedToF depth camera, the depth errors can be either negative or positive,because the light leakage can cause the phase to shift either forward orbackward. For example, a depth map for even a flat wall in theenvironment can include sinusoidal ripples due the light leakage.Furthermore, if the light leakage is sufficiently strong, additionaldepth errors can occur during a dealiasing process, during which resultsof multiple frequency measurements are compared to extract the depthinformation. Similar to the pulse-based ToF depth camera, the lightleakage mask can help eliminate or reduce the depth error due to thelight leakage in phase-based ToF depth camera as well.

The structured light stereo matching system can also use the lightleakage mask for achieving accurate depth measurement. The structuredlight stereo matching system uses structured light pattern to measuredepth via triangulation. The system performs stereo matching between thecaptured camera image and a virtual image that corresponds to thestructured light pattern of the illumination module. If the capturedcamera image is contaminated by the depth error due to the lightleakage, those two images are more dissimilar from each other. As aresult, the stereo matching process more likely produces inaccuratematches, leading to inaccurate depth values. By subtracting the lightleakage mask from the captured camera image, the stereo matching betweenthe captured image (after the subtraction) and the virtual image resultsin more accurate depth values.

Generation of Light Leakage Mask During Camera Calibration

In some embodiments, the light leakage mask can be generated offline,e.g., during camera calibration, since the light leakage mask isconstant and does not depend on movement of the depth sensing device ormovement of the object to be measured. For example, the light leakagemask can be generated during a camera calibration process when the depthsensing device is manufactured and calibrated.

FIG. 5 illustrates a sample process of generating a light leakage maskduring camera calibration. At step 510 of the process 500, the depthsensing device (including the illumination module, the depth camera andthe optical component) facing an empty space initiates a calibrationprocess. Here an empty space refers to a space where the environment issufficiently far from the depth camera such that the depth camera doesnot receive a meaningful amount of light signals that are emitted by theillumination module and reflected by a surface of the environment.Furthermore, there is no ambient light in the empty space.

An instance of the empty space is a night sky without any stars or moon.In some embodiments, the empty space can be set up such that there isenough space between the depth sensing device and a wall. The wall has alow reflectivity such that the depth camera only receives a negligibleamount of light signals that are emitted by the illumination module andreflected by a surface of the wall. Ambient light sources (e.g., windowor lamp) are also avoided or switched off.

At step 515, the depth sensing device turns on the illumination module.Since the depth camera does not receive any light signals that arereflected by the surface of the environment, the only light signals thatthe depth camera receives are the light signals that are emitted by theillumination module and redirected by the optical component.

At step 520, the depth camera records multiple empty space images thatcapture the leaked light redirected by the optical component. At step525, the depth sensing device generates the light leakage mask bycalculating a temporal average image of the multiple empty space images.The advantage of calculating an average of multiple empty space imagesis to eliminate the photon noise from the light leakage mask.

Real-Time Generation of Light Leakage Mask

Instead of generating the light leakage mask during camera calibration,the depth sensing device can also generate the light leakage mask whilethe depth sensing device is operating (on-the-fly). The assumption isthat the depth camera and/or the object to be measured (i.e.,environment) are moving such that each pixel of the depth cameraobserves an empty space at least at certain time points. One advantageof generating the light leakage mask on-the-fly over generating the maskduring camera calibration is that the on-the-fly generation of the lightleakage mask takes into consideration any changes to the opticalcomponent after the system is calibrated. For example, the light leakagemask generated on-the-fly can capture effect of fingerprintcontamination on the visor (or other types of optical components), onwhich a user's hand causes the fingerprint.

FIG. 6 illustrates a sample process of generating a light leakage maskon-the-fly. For the process 600, the assumption is that the each pixelof a moving depth camera observes empty space for at least apredetermined percentage (e.g., 5%) of the time. At step 605 of theprocess 600, the depth camera of the depth sensing device capturesmultiple images over a time period. At step 610, for each pixel of thedepth camera, the depth sensing device identifies a predeterminedpercentage (e.g., 5%) of images that have the lowest captured intensityvalues (also referred to as responses) for that pixel over the timeperiod.

At step 615, the depth sensing device generates a pixel value for acorresponding pixel of the light leakage mask by calculating an averagevalue of the identified lowest captured intensity values. The purpose ofaveraging is to illuminate the photo noise for that pixel. At step 620,the depth sensing device generates the light leakage mask by groupingthe pixel values into an image.

FIG. 7 illustrates another sample process of generating a light leakagemask on-the-fly. The process 700 is not based on the assumption thateach pixel observes empty space for certain amount of time. Instead, theprocess 700 actively tries to detect empty space observations based onone or more constraints. At step 705 of the process 700, the depthcamera of the depth sensing device captures an image. At step 710, thedepth sensing device selects a pixel of the captured image.

At step 715, the depth sensing device determines whether the selectedpixel has a valid depth reading. A valid depth reading refers to a depthreading that is within the hardware limit of the camera pixel. If thepixel has a valid depth reading, the process 700 proceeds to step 720.If the pixel does not have a valid depth reading, this suggests that thepixel may observe an empty space and the process 700 proceeds to step725.

At step 720, the depth sensing device determines whether the valid depthreading is close to the depth of the optical component (e.g., visor).The depth of the optical component refers to the distance between theoptical component and the depth camera. The closeness can be determinedbased on a threshold value. If a difference between the pixel depthreading and the depth of the optical component is less than thethreshold value, the depth reading is determined to be close to thedepth of the optical component. Such a close depth reading also suggeststhat the pixel may observe an empty space. If so, the process 700proceeds to step 725. Otherwise the process 700 proceeds to step 740.

At step 725, the depth sensing device determines whether the signal issaturated on the pixel of the sensor. For example, the signal issaturated if the received light signal is significantly above a leakageexpectation (based on, e.g., a threshold value). If the signal issaturated, the process 700 proceeds to step 740. If the signal is notsaturated, the process 700 proceeds to step 730, wherein the depthsensing device identifies the pixel as being observing an empty space.At step 735, the depth sensing device records the value of the pixel aspart of the light leakage mask. Then the process 700 proceeds to step740.

At step 740, the depth sensing device determines whether all pixels areexamined. If so, the process 700 proceeds to step 705 to capture anotherimage. Otherwise, the process 700 proceeds to step 710 to select anotherpixel for examination.

The process 700 illustrates constraints for detecting empty spaceobservation and pixel depth reading for the light leakage mask, such asdepth reading validness (at step 715), closeness to optical component(at step 720), and signal saturation (at step 725). In some otherembodiments, the depth sensing device can identify pixel values for thelight leakage mask based on other constraints.

For example, if depth sensing device is a pulse-based ToF depth camera,the depth sensing device can examine shutter images that not subject tolight leakage as a criterion to determine whether a pixel observes anempty space. Multiple shutter images are available to the pulse-basedToF depth camera because the shutter can be opened and closed formultiple times since a point in time when the illumination module emitsa light signal. The time periods when the shutter opens are calledshutter windows. By adjusting the opening and closing times of theshutter, the depth camera records multiple shutter images at differenttime windows. FIG. 8 shows intensities observed in four shutter images.The readings from the multiple images are used to eliminate effects ofunknown variables such as environment reflectivity and ambient lightintensity, and to provide robustness against multi-path interference.

As shown in FIG. 8, unlike the 1st shutter window 810, the opening timesof the 2nd, 3rd and 4th shutter windows (820, 830, 840) are after thetime point when the emitted light travels for 628 mm and then reachesthe depth camera. Due to the short distance between the opticalcomponent (e.g., visor) and the depth camera (which is significantlyless than 628 mm), the leaked light has already reached the depth camerabefore the 2nd, 3rd and 4th shutter windows (820, 830, 840). In otherwords, the 2nd, 3rd and 4th shutter images are not subject to lightleakage due to the light redirection of the optical component. If thecorresponding pixel of any of the 2nd, 3rd and 4th shutter imagescontains light signals, the depth sensing device can determine that thatpixel is not observing an empty space.

In some embodiments, the depth sensing device can use a machine learningprocess to train a classifier based on a training set with verifiedclassification data for determining whether a pixel of a captured imageobserves an empty space and qualifies as part of the light leakage mask.For example, such a machine learning classifier can be based on, e.g.,neural networks or random decision forests.

Reduction of Depth Error by Controlling Shutter

Instead of using a light leakage mask, the depth sensing device can alsocontrol the shutter operation to block light signals (i.e., photon) thatare redirected by the optical component (e.g., visor). Particularly fora pulse-based ToF depth camera, the system can adjust the opening andclosing time points of shutter windows of the shutter such that no lightsignals that travel for less than a threshold value reach the depthcamera during any of the shutter windows. If the threshold value exceedsor equals a distance for which the leaked light travels from theillumination module to the optical component and then reaches the depthcamera, the depth camera captures no leaked light that causes the deptherror.

FIG. 9A shows shutter windows of a pulse-based ToF depth camera that arenot adjusted to avoid capturing leaked light. The Y-axis represents thelight intensities of the shutter images captured during the shutterwindows. The X-axis represents the corresponding depth readings.Assuming the depth of the optical component (e.g., visor) is 5 cm, theshutter images 910 and 920 collect a large number of photons around thedepth of the optical component. Therefore, the depth readings based onthe shutter images 910, 920, 930 and 940 are heavily affected by theleaked light due to the optical component.

FIG. 9B shows shutter windows of a pulse-based ToF depth camera that areadjusted to avoid capturing leaked light. As illustrated in FIG. 9B, asubstantially reduced number of photons are collected around the closedepth of the optical component by the shutter images 925, 935 and 945.Even the shutter image 915 only collects a small number of photonsaround the depth of the optical component. Thus, the light leakagecontamination is significantly reduced. Therefore, the depth sensingdevice can reduce or eliminate the depth error by controlling theshutter windows. The leaked light can be avoided if no photons or asmall number of photons at the depth of the optical component iscollected by the shutter images. Such a controlling of the shutteroperation can be applied to avoid light leakage due to manufacturingimperfections of the protective optical component, as well as foreignobjects (e.g., a smudge or fingerprint) on the optical component,

Sample Hardware Architecture

FIG. 10 shows a high-level example of a hardware architecture of aprocessing system that can be used to implement the disclosed functions.The processing system illustrated in FIG. 10 can be, e.g., a subsystemof the HMD device, the NED device or other depth sensing devices. One ormultiple instances of an architecture such as shown in FIG. 10 (e.g.,multiple computers) can be used to implement the techniques describedherein, where multiple such instances can be coupled to each other viaone or more networks.

The illustrated processing system 1000 includes one or more processors1010, one or more memories 1011, one or more communication device(s)1012, one or more input/output (I/O) devices 1013, and one or more massstorage devices 1014, all coupled to each other through an interconnect1015. The interconnect 1015 may be or include one or more conductivetraces, buses, point-to-point connections, controllers, adapters and/orother conventional connection devices. Each processor 1010 controls, atleast in part, the overall operation of the processing device 1000 andcan be or include, for example, one or more general-purpose programmablemicroprocessors, digital signal processors (DSPs), mobile applicationprocessors, microcontrollers, application specific integrated circuits(ASICs), programmable gate arrays (PGAs), or the like, or a combinationof such devices.

Each memory 1011 can be or include one or more physical storage devices,which may be in the form of random access memory (RAM), read-only memory(ROM) (which may be erasable and programmable), flash memory, miniaturehard disk drive, or other suitable type of storage device, or acombination of such devices. Each mass storage device 1014 can be orinclude one or more hard drives, digital versatile disks (DVDs), flashmemories, or the like. Each memory 1011 and/or mass storage 1014 canstore (individually or collectively) data and instructions thatconfigure the processor(s) 1010 to execute operations to implement thetechniques described above. Each communication device 1012 may be orinclude, for example, an Ethernet adapter, cable modem, Wi-Fi adapter,cellular transceiver, baseband processor, Bluetooth or Bluetooth LowEnergy (BLE) transceiver, or the like, or a combination thereof.Depending on the specific nature and purpose of the processing system1000, each I/O device 1013 can be or include a device such as a display(which may be a touch screen display), audio speaker, keyboard, mouse orother pointing device, microphone, camera, etc. Note, however, that suchI/O devices may be unnecessary if the processing device 1000 is embodiedsolely as a server computer.

In the case of a user device, a communication device 1012 can be orinclude, for example, a cellular telecommunications transceiver (e.g.,3G, LTE/4G, 5G), Wi-Fi transceiver, baseband processor, Bluetooth or BLEtransceiver, or the like, or a combination thereof. In the case of aserver, a communication device 1012 can be or include, for example, anyof the aforementioned types of communication devices, a wired Ethernetadapter, cable modem, DSL modem, or the like, or a combination of suchdevices.

The machine-implemented operations described above can be implemented atleast partially by programmable circuitry programmed/configured bysoftware and/or firmware, or entirely by special-purpose circuitry, orby a combination of such forms. Such special-purpose circuitry (if any)can be in the form of, for example, one or more application-specificintegrated circuits (ASICs), programmable logic devices (PLDs),field-programmable gate arrays (FPGAs), system-on-a-chip systems (SOCs),etc.

Software or firmware to implement the embodiments introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium,” as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

Examples of Certain Embodiments

Certain embodiments of the technology introduced herein are summarizedin the following numbered examples:

1. An method of depth sensing including: emitting light, by a lightemitter, to illuminate an environment of a depth sensing device;identifying a first portion of the emitted light that is prevented fromreaching the environment of the depth sensing device due to beingredirected by an optical component located in proximity to the lightemitter; detecting, by an imaging sensor of the depth sensing device, asecond portion of the emitted light that reaches and is reflected by asurface in the environment of the depth sensing device other than asurface of the optical component; and generating, based on the secondportion of the emitted light, a depth map that includes a plurality ofpixel values corresponding to distances relative to the depth sensingdevice, wherein said generating excludes from consideration theidentified first portion of the emitted light.

2. The method of example 1, further including: generating a lightleakage mask including pixel values corresponding to the first portionof the emitted light that is prevented from reaching the environment ofthe depth sensing device due to being redirected by the opticalcomponent, wherein the light leakage mask improves an accuracy of thedepth map; and detecting an object in the environment of the depthsensing device based on the depth map.

3. The method of example 2, wherein said generating of the depth mapincludes: recording, by the imaging sensor, an image based on thedetected light; generating a processed image by subtracting the lightleakage mask from the recorded image; and converting the processed imageto the depth map that includes the pixel values corresponding to depthsof the environment relative to the depth sensing device.

4. The method of example 3, wherein the light emitted by the lightemitter includes a pulse of light, and wherein said converting includesconverting a pixel value of the depth map based on a time of flight fromthe light emitter to the environment and then to the imaging sensor.

5. The method of example 3 or 4, wherein the light emitted by the lightemitter is frequency-modulated, and wherein said converting includesconverting a pixel value of the depth map based on a phase differencebetween the light emitted by the light emitter and the light detected bythe imaging sensor.

6. The method in any of the preceding examples 3 through 5, wherein thelight emitted by the light emitter has a structured light pattern, andwherein said converting includes converting a pixel value of the depthmap by stereo matching between the processed image and a virtual imagethat corresponds to the structured light pattern.

7. The method in any of the preceding examples 1 through 6, wherein thefirst portion of the emitted light is redirected by a foreign object onthe optical component or due to a manufacturing imperfection of theoptical component.

8. The method in any of the preceding examples 1 through 7, furtherincluding: performing a calibrating process by the depth sensing devicefacing an empty space; and generating the light leakage mask includingpixel values corresponding light that is redirected by the opticalcomponent and reaches the depth camera.

9. The method in any of the preceding examples 1 through 8, furtherincluding: identifying a pixel that observes an empty space when thedepth sensing device is in operation; and generating the light leakagemask including a pixel value of the identified pixel.

10. The method in any of the preceding examples 1 through 9, furtherincluding: closing a shutter of the imaging sensor during a first timeperiod when the first portion of the emitted light redirected by theoptical component is reaching the shutter; and opening the shutterduring a second time period when the imaging sensor receives through theoptical component the second portion of the emitted light reflected bythe surface in the environment.

11. A depth sensing device including: an illumination module that, whenin operation, emits light, wherein a first portion of the emitted lightilluminates an environment of the depth sensing device; an opticalcomponent located in proximity to the illumination module, wherein theoptical component redirects and prevents a second portion of the emittedlight from reaching the environment of the depth sensing device; animaging sensor that, when in operation, receives the first and secondportions of the emitted light and records an image based on the receivedlight; and a processor that, when in operation, generates a processedimage by subtracting a light leakage mask from the recorded image, thelight leakage mask including pixel values corresponding to the secondportion of the emitted light that is prevented from reaching theenvironment of the depth sensing device due to being redirected by theoptical component, and converts the processed image into a depth mapthat includes a plurality of pixel values corresponding to depths of theenvironment relative to the depth sensing device.

12. The depth sensing device of example 11, wherein the depth sensingdevice performs a calibration process when the depth sensing devicefaces an empty space, and the processor generates the light leakage maskduring the calibration process.

13. The depth sensing device of example 11 or 12, wherein the processor,when in operation, identifies a pixel that observes an empty space whenthe depth sensing device is in operation, and generates the lightleakage mask including a pixel value of the identified pixel.

14. The depth sensing device in any of the preceding examples 11 through13, wherein the imaging sensor, when in operation, records multipleimages; and wherein the processor, when in operation, identifies apercentage of the images that have the lowest captured intensity valuesfor an individual pixel, and generates a pixel value of the lightleakage mask by calculating an average value of the identified lowestcaptured intensity values for the individual pixel.

15. The depth sensing device in any of the preceding examples 11 through14, wherein the processor, when in operation, identifies a pixel thatdoes not have a valid depth reading or has a depth reading that is closeto a depth of the optical component within a threshold value, andgenerates the light leakage mask including a pixel value of theidentified pixel.

16. The depth sensing device of example 15, wherein the light is notsaturated at the identified pixel of the imaging sensor.

17. The depth sensing device of example 15 or 16, wherein the identifiedpixel of a shutter image that is not subject to light leakage does notcontain a light signal.

18. A depth sensing device including: an optical component; anillumination module located in proximity to the optical component, whenin operation, emits light towards an environment of the depth sensingdevice, wherein a first portion of the emitted light is prevented fromreaching the environment due to being redirected by the opticalcomponent, and a second portion of the emitted light illuminates theenvironment and is reflected through the optical component by a surfacein the environment other than a surface of the optical component; and animagining sensor including a shutter, wherein the shutter, when inoperation, closes during a first time period when the first portion ofthe emitted light redirected by the optical component is reaching theshutter, and the shutter, when in operation, opens during a second timeperiod when the imaging sensor receives through the optical componentthe second portion of the emitted light reflected by the surface of theenvironment.

19. The depth sensing device of example 18, wherein opening and closingoperations of the shutter are controlled to prevent the imaging sensorfrom receiving the first portion of the emitted light that is preventedfrom reaching the environment due to being redirected by the opticalcomponent.

20. The depth sensing device of example 18 or 19, wherein there aremultiple shutter windows since the illumination module emits a pulse oflight, and a first shutter window among the multiple shutter windowsopens after the pulse of light reaches the imaging sensor.

21. An apparatus of depth sensing including: means for emitting light,by a light emitter, to illuminate an environment of a depth sensingdevice; means for identifying a first portion of the emitted light thatis prevented from reaching the environment of the depth sensing devicedue to being redirected by an optical component located in proximity tothe light emitter; means for detecting, by an imaging sensor of thedepth sensing device, a second portion of the emitted light that reachesand is reflected by a surface in the environment of the depth sensingdevice other than a surface of the optical component; and means forgenerating, based on the second portion of the emitted light, a depthmap that includes a plurality of pixel values corresponding to distancesrelative to the depth sensing device, wherein said generating excludesfrom consideration the identified first portion of the emitted light.

22. The apparatus of example 21, further including: means for generatinga light leakage mask including pixel values corresponding to the firstportion of the emitted light that is prevented from reaching theenvironment of the depth sensing device due to being redirected by theoptical component, wherein the light leakage mask improves an accuracyof the depth map; and means for detecting an object in the environmentof the depth sensing device based on the depth map.

23. The apparatus of example 22, wherein said generating of the depthmap includes: means for recording, by the imaging sensor, an image basedon the detected light; means for generating a processed image bysubtracting the light leakage mask from the recorded image; and meansfor converting the processed image to the depth map that includes thepixel values corresponding to depths of the environment relative to thedepth sensing device.

24. The apparatus of example 23, wherein the light emitted by the lightemitter includes a pulse of light, and wherein said converting includesconverting a pixel value of the depth map based on a time of flight fromthe light emitter to the environment and then to the imaging sensor.

25. The apparatus of example 23 or 24, wherein the light emitted by thelight emitter is frequency-modulated, and wherein said convertingincludes converting a pixel value of the depth map based on a phasedifference between the light emitted by the light emitter and the lightdetected by the imaging sensor.

26. The apparatus in any of the preceding examples 3 through 5, whereinthe light emitted by the light emitter has a structured light pattern,and wherein said converting includes converting a pixel value of thedepth map by stereo matching between the processed image and a virtualimage that corresponds to the structured light pattern.

27. The apparatus in any of the preceding examples 21 through 26,wherein the first portion of the emitted light is redirected by aforeign object on the optical component or due to a manufacturingimperfection of the optical component.

28. The apparatus in any of the preceding examples 21 through 27,further including: means for performing a calibrating process by thedepth sensing device facing an empty space; and means for generating thelight leakage mask including pixel values corresponding light that isredirected by the optical component and reaches the depth camera.

29. The apparatus in any of the preceding examples 21 through 28,further including: means for identifying a pixel that observes an emptyspace when the depth sensing device is in operation; and means forgenerating the light leakage mask including a pixel value of theidentified pixel.

30. The apparatus in any of the preceding examples 21 through 29,further including: means for closing a shutter of the imaging sensorduring a first time period when the first portion of the emitted lightredirected by the optical component is reaching the shutter; and meansfor opening the shutter during a second time period when the imagingsensor receives through the optical component the second portion of theemitted light reflected by the surface in the environment.

Any or all of the features and functions described above can be combinedwith each other, except to the extent it may be otherwise stated aboveor to the extent that any such embodiments may be incompatible by virtueof their function or structure, as will be apparent to persons ofordinary skill in the art. Unless contrary to physical possibility, itis envisioned that (i) the methods/steps described herein may beperformed in any sequence and/or in any combination, and that (ii) thecomponents of respective embodiments may be combined in any manner.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims, and other equivalent features and acts are intended to be withinthe scope of the claims.

What is claimed is:
 1. A method of depth sensing, comprising: emittinglight, by a light emitter, to illuminate an environment of a depthsensing device; identifying a first portion of the emitted light that isprevented from reaching the environment of the depth sensing device dueto being redirected; detecting, by an imaging sensor of the depthsensing device, a second portion of the emitted light that reaches andis reflected by a surface in the environment of the depth sensing deviceother than a surface of an optical component located in proximity to thelight emitter; and generating, based on the second portion of theemitted light, a depth map that includes a plurality of pixel valuescorresponding to distances between the environment and the depth sensingdevice without any depth error caused by deviations due to the firstportion of the emitted light, wherein the generating excludes fromconsideration the identified first portion of the emitted light.
 2. Themethod of claim 1, further comprising: generating a light leakage maskincluding pixel values corresponding to the first portion of the emittedlight that is prevented from reaching the environment of the depthsensing device due to being redirected by the optical component, whereinthe light leakage mask improves an accuracy of the depth map; anddetecting an object in the environment of the depth sensing device basedon the depth map.
 3. The method of claim 2, wherein said generating ofthe depth map includes: recording, by the imaging sensor, an image basedon the detected light; generating a processed image by subtracting thelight leakage mask from the recorded image; and converting the processedimage to the depth map that includes the pixel values corresponding todepths of the environment relative to the depth sensing device.
 4. Themethod of claim 3, wherein the light emitted by the light emitterincludes a pulse of light, and wherein said converting includesconverting a pixel value of the depth map based on a time of flight fromthe light emitter to the environment and then to the imaging sensor. 5.The method of claim 3, wherein the light emitted by the light emitter isfrequency-modulated, and wherein said converting includes converting apixel value of the depth map based on a phase difference between thelight emitted by the light emitter and the light detected by the imagingsensor.
 6. The method of claim 3, wherein the light emitted by the lightemitter has a structured light pattern, and wherein said convertingincludes converting a pixel value of the depth map by stereo matchingbetween the processed image and a virtual image that corresponds to thestructured light pattern.
 7. The method of claim 1, wherein the firstportion of the emitted light is redirected by a foreign object on theoptical component or due to a manufacturing imperfection of the opticalcomponent.
 8. The method of claim 1, further comprising: performing acalibrating process by the depth sensing device facing an empty space;and generating the light leakage mask including pixel valuescorresponding light that is redirected by the optical component andreaches the depth camera.
 9. The method of claim 1, further comprising:identifying a pixel that observes an empty space when the depth sensingdevice is in operation; and generating the light leakage mask includinga pixel value of the identified pixel.
 10. The method of claim 1,further comprising: closing a shutter of the imaging sensor during afirst time period when the first portion of the emitted light redirectedby the optical component is reaching the shutter; and opening theshutter during a second time period when the imaging sensor receivesthrough the optical component the second portion of the emitted lightreflected by the surface in the environment.
 11. A depth sensing devicecomprising: an illumination module having a lighting device that, whenin operation, emits light, wherein a first portion of the emitted lightilluminates an environment of the depth sensing device; an opticalcomponent located in proximity to the illumination module, wherein theoptical component redirects and prevents a second portion of the emittedlight from reaching the environment of the depth sensing device; animaging sensor that, when in operation, receives the first and secondportions of the emitted light and records an image based on the receivedlight; and a processor that, when in operation, generates a processedimage by subtracting a light leakage mask from the recorded image, thelight leakage mask including pixel values corresponding to the secondportion of the emitted light that is prevented from reaching theenvironment of the depth sensing device due to being redirected by theoptical component, and converts the processed image into a depth mapthat includes a plurality of pixel values corresponding to distancesbetween the environment and the depth sensing device without any deptherror caused by deviations due to the second portion of the emittedlight.
 12. The depth sensing device of claim 11, wherein the depthsensing device performs a calibration process when the depth sensingdevice faces an empty space, and the processor generates the lightleakage mask during the calibration process.
 13. The depth sensingdevice of claim 11, wherein the processor, when in operation, identifiesa pixel that observes an empty space when the depth sensing device is inoperation, and generates the light leakage mask including a pixel valueof the identified pixel.
 14. The depth sensing device of claim 11,wherein the imaging sensor, when in operation, records multiple images;and wherein the processor, when in operation, identifies a percentage ofthe images that have the lowest captured intensity values for anindividual pixel, and generates a pixel value of the light leakage maskby calculating an average value of the identified lowest capturedintensity values for the individual pixel.
 15. The depth sensing deviceof claim 11, wherein the processor, when in operation, identifies apixel that does not have a valid depth reading or has a depth readingthat is close to a depth of the optical component within a thresholdvalue, and generates the light leakage mask including a pixel value ofthe identified pixel.
 16. The depth sensing device of claim 15, whereinthe light is not saturated at the identified pixel of the imagingsensor.
 17. The depth sensing device of claim 15, wherein the identifiedpixel of a shutter image is not subject to light leakage due to theoptical component redirecting and preventing the second portion of theemitted light from reaching the environment of the depth sensing device.18. A depth sensing device comprising: an optical component; anillumination module located in proximity to the optical component andhaving a lighting device that, when in operation, emits light towards anenvironment of the depth sensing device, wherein a first portion of theemitted light is prevented from reaching the environment due to beingredirected by the optical component, and a second portion of the emittedlight illuminates the environment and is reflected through the opticalcomponent by a surface in the environment other than a surface of theoptical component; an imaging sensor including a shutter, wherein theshutter, when in operation, closes during a first time period when thefirst portion of the emitted light redirected by the optical componentis reaching the shutter, and the shutter, when in operation, opensduring a second time period when the imaging sensor receives through theoptical component the second portion of the emitted light reflected bythe surface of the environment; and a processor, when in operation,generates a depth map that includes a plurality of pixel valuescorresponding to distances between the environment and the depth sensingdevice without any depth error caused by deviations due to the firstportion of the emitted light.
 19. The depth sensing device of claim 18,wherein opening and closing operations of the shutter are controlled toprevent the imaging sensor from receiving the first portion of theemitted light that is prevented from reaching the environment due tobeing redirected by the optical component.
 20. The depth sensing deviceof claim 18, wherein there are multiple shutter windows since theillumination module emits a pulse of light, and a first shutter windowamong the multiple shutter windows opens after the pulse of lightreaches the imaging sensor.